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(54) An image processing system 

(57) Four images are acquired from overlapping 
fields of view, and are combined to provide a single im- 



age of higher resolution. The camera and the object are 
displaced relative to each other in short moves of one 
half a pixel. 



Step 1 Acquire first image 




Scene Under Examination 

Fig. 1(a) 
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Step 2 Acquire second Image 



1/2 Pixel Move 




Fig. 1(b) 
Step 3 Acquire third image 



1 12 Pixel Move 
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Fig. 1(c) 

Step 4 Acquire fourth image 



1/2 Pixel Move 




Fig. 1(d) 
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Description 

Introduction 

[0001] The invention relates to capture and process- 5 
ing of images from a subject such as an electronics cir- 
cuit board. 

[0002] At present, it is known to provide an image 
processing system comprising a camera such as a CCD 
camera, and an XY table with a gantry. The gantry 10 
moves either the subject or the camera. For example, 
the camera may have a resolution of 1 280 x 1 024 pixels, 
for a pixel size of 25 microns resolution. This gives a 
field of view of 32 x 25 mm. Thus, 1 00 views are required 
to inspect a circuit board of size 320 mm x 250 mm. The 15 
camera may have an acquisition time of, say, 50 msec. 
If the time for movement from view to view is on average 
200 ms, the total time per view is 250 ms, giving four 
views per second and thus a total time of 25 sees for the 
full board (1 00 views). 20 
[0003] If the board is to be inspected at 12.5 micron 
resolution, the prior approach has been to change the 
optics to reduce the pixel size to 1 2.5 microns. Thus 400 
views are required. The camera acquisition will still take 
50 msec, and the move time will reduce to 140 msec, 25 
allowing 5.2 views / second to be acquired. Thus the 
board can be inspected in 77 sec, more than 3 times 
longer than at 25 micron pixel size. In addition, there is 
the problem of having to split up large components that 
are greater than 16x 12.5 mm but less then 32x25 mm. 3D 
[0004] The invention is therefore directed towards 
achieving a shorter time for full capture over a given area 
at higher resolution. The invention furthermore aims to 
achieve these objectives without needing to modify a 
conventional optical set-up of an inspection machine. 35 

Statements of Invention 

[0005] According to the invention, there is provided 
an image processing system comprising a controller, a *o 
camera, movement means to change effective location 
of the camera with respect to a target object , and means 
for processing the captured images, wherein 



of higher resolution than that of the initial images. 

[0006] In one embodiment, the initial images are mu- 
tually offset by a fraction of a pixel width. 
[0007] In another embodiment, the initial images are 55 
offset by one half a pixel in X and Y directions. 
[0008] In a further embodiment, the initial images are 
offset by a third of a pixel in X and Y directions. 



[0009] In one embodiment, the invention provides the 
further step of interpolating additional pixels from the 
captured initial images. 

[0010] In another embodiment, the controller com- 
prises means for determining for which parts of an object 
multiple offset initial images are to be captured, whereby 
view images of differing resolutions are obtained for an 
object. 

[0011] In a further embodiment, the controller com- 
prises means for directing capture of initial images of 
different colours for a particular view image. 
[0012] In one embodiment, the movement means 
comprises means for moving a lens with respect to the 
remainder of the camera. 

[0013] In another embodiment, the movement means 
comprises means for moving the lens in X and Y direc- 
tions. 

[0014] In one embodiment, the movement means 
comprises means for tilting the lens. 

Detailed Description of the Invention 

[0015] The invention will be more clearly understood 
from the following description of some embodiments 
thereof, given by way of example only with reference to 
the accompanying drawings in which:- 

Figs. 1 (a), 1 (b), 1 (c) and 1 (d) are diagrams illustrat- 
ing a camera movement sequence for acquisition 
of four images 

Figs. 2(a), 2(b), 2(c) and 2(d) are diagrams illustrat- 
ing the four images captured during the movement 
sequence 1(a) to 1(d). 

Figs. 3(a) and 3(b) illustrate how a super-resolution 
image is assembled using pixels from the images 2 
(a) to 2(d). 

Fjgs. 4(a) and 4(b) are diagrams illustrating a cam- 
era movement sequence for acquisition of 2 imag- 
es.. 

Figs. 5(a) and 5(b) are diagrams illustrating the two 
images captured during the movement sequence 4 
(a) and 4(b). 

Fig. 6(a) illustrates how a super-resolution image is 
formed using pixels from image 5(a) and 5(b), and 
Fig. 6(b) illustrates a further step of interpolation of 
missing pixels. 

Fig. 7 is a diagram illustrating a camera movement 
sequence; 

Figs. 8(a) and 8(b) are diagrams illustrating photo- 
sensitive areas for image capture; and 



40 



the camera and the movement means comprise 45 
means for capturing a plurality of mutually offset in- 
itial images for a view, and 

the image processing means comprises means for 
combining the initial images to provide a view image 50 
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Figs. 9 to 1 2 are diagrams illustrating further move- 
ment sequences. 

[0016] In one embodiment, a 4-image-super-resolu- 
tion image Is produced. 

[0017] The imaging system comprises a camera em- 
ploying an image sensor that is made up of a finite 
number of imaging elements. When imaging an object 
these pixels will have a finite resolution known as the 
pixel size. For a non-zoom optical system this is fixed. 
The system further comprises an x-y table with a gantry. 
The gantry moves either the object or the camera. It is 
only possible to produce a super-resolution image if the 
inspection system i.e. the camera or object is capable 
of movements which are less than one tenth of the pixel 
size. 

[0018] Figs. 1 to 3 illustrate the method for acquisition 
and generation of a super-resolution image based on 
four images. In this example four images are acquired 
from overlapping fields of view (FOV) and combined to 
produce a single super resolution image. The camera 
and object are displaced relative to each other in three 
short moves. The resultant image will have four times 
the resolution of the original camera image. It has twice 
the linear resolution of the original camera image. 
[001 9] Referring to Figs. 1 (a) to 1 (d) a camera move- 
ment sequence for acquisition of four images is illustrat- 
ed. Figs. 1 (a) to 1 (d) show the FOV of the scene under 
examination, Fig 1 (a) illustrates the initial FOV and Figs 
1(b) to 1 (d) show the FOV after movement. 
[0020] The following steps 1 to 4 of the sequence cor- 
respond to Figs. 1 (a) to 1 (d) respectively. Step 5 is illus- 
trated by Figs. 3(a) and 3(b). 

Step 1 : Acquire and store first image. 

Step 2: Move the camera left by % of a pixel width. 
Acquire and store the second image. 



shown in the Image Co-ordinate Reference. 
The insertion position of each pixel is made 
according to the move sequence used to ac- 
quire the images. This pattern is repeated un- 
5 til ail pixel locations in the super-resolution 

image are filled. 

[0022] In a further embodiment, a method is de- 
scribed whereby it is possible to create an image with 
10 four times the pixel count of the original while using only 
two captured images. 

[0023] Figs. 4 to 6 illustrate a method for generation 
of a two-image super-resolution image. This method 
has the advantage that is twice as quick as the previ- 
15 ously described four-image method. Figs. 4(a) and 4 (b) 
illustrate the sequence of movement for capturing two 
images. The process involves the following steps: 

Step 1 : Acquire and store first image. 
20 step 2: Move the camera left by !4 a pixel width and 
down by a pixel width Acquire and store 
second image. 

[0024] At this point the inspection system memory will 
25 contain two images, images 1 and 2 as illustrated in 
Figs. 5(a) and 5 (b). 

[0025] The method for creating the super-resolution 
image is illustrated by Figs 6(a) and 6(b). This involves 
the following steps: 

30 

Step 3: The super resolution images is populated. 

Pixels are inserted from images 5(a) and 5 
(b) according to a checker board pattern as 
shown in Fig. 6(a). Starting at the upper left- 
35 hand corner (0,0), firstly pixels from image 5 

(a) are inserted and next the pixels from im- 
age 5(b) are inserted until all pixel positions 
are filled. 



30 



Step 3: Move the cameFa down by % of a pixel width. 
Acquire and store third image. 

Step 4: Move the camera right by 3£ of a pixel width. 
Acquire and store fourth image. 

[0021] The memory of the inspection system will now 
contain four images. The four captured images 1,2,3 
and 4 are illustrated in Figs. 2(a) to 2(d). 

Step 5: Relates to the composition of a four-image 
super-resolution image. Each individual pixel 
in each of the images 1,2,3 and 4 becomes 
a pixel in the super-resolution image. Refer- 
ring to Figs. 3 (a) and 3(b), the method for 
composition of the super-resolution image is 
as follows. Starting at the upper left-hand cor- 
ner, (0,0), pixels are inserted into the super- 
resolution images according to the sequence 



[0026] In the method of this embodiment, which is 
based on the use of two overlapping images, it is noted, 
as illustrated in Fig 6(a) that half of the image contains 
no original image data. The missing pixels are interpo- 
lated as follows. 

Step 4: As illustrated in Fig. 6(b), the missing pixels 
are interpolated from the surrounding north, 
south, east and west pixels. The interpolated 
pixel is then placed in the empty location. 

[0027] The inspection system is not limited to the 
methods of acquisition of 2 or 4 images to create a su- 
per-resolution image. For example, if the camera move- 
ment system is capable of sufficiently fine movements, 
then the system may acquire 9 images (using moves of 
1/3 of a pixel width) in order to produce a super resolu- 
tion of three times the linear resolution of the original. 
[0028] Further embodiments of the system are illus- 
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trated with reference to Figs. 7 to 12. 
[0029] In one embodiment, referring back to the ex- 
ample outlined in the Introduction, the optical resolution 
is left at 25 microns. The camera is controlled to acquire 
four images at each "location", each 12.5 microns apart 
in X and Y. The four 25 micron resolution images are 
combined to form a single 12.5 micron image. 
[0030] The view-to-view move time is unchanged at 
200 msec, while the acquisition time is increased to 200 
msec as we have to acquire 4 images. We can perform 
the small moves in less than the sensor readout time 
( 45 msec ), and thus this has no effect on the total im- 
age acquisition time. Thus we can perform 2.5 views / 
second and inspect the board in 40 seconds, almost 
twice as fast as the prior "optical high resolution" situa- 
tion. 

[0031] Let us call the figure of 12.5 microns HX ( and 
also HY) for Half X pixel ( and Half Y pixel ). The move 
sequence is as set out in Fig. 7. 

[0032] The moves are carried out in the sequence A 
BCD. 

[0033] If the camera is twice as fast ( 25 msec to ac- 
quire an image ), we can inspect the board in 30 sec- 
onds or 3.3 views /second or 2.2 times as fast as the 
high resolution case. 

[0034] There are three further advantages: 

A: We can inspect larger parts without "breaking 
them up" between views which allows more robust 
vision algorithms to be used. 

B: We do not need to modify the system hardware 
to achieve this increase in resolution, and thus it is 
easier to upgrade existing machines. 

C: We can mix high and normal resolution views in 
the same inspection program in any order. Suppose 
we only had small parts in 20 percent of the views 
on the board, we would only have to acquire the ex- 
tra images for these views, further speeding up the 
system. 

We can consider this a "mechanical zoom" system, 
which can be added to a system by a software ad- 
dition. 

[0035] The performance of the system is further en- 
hanced by the "fill factor" of a CCD sensor when it is 
less than 100 %. 

[0036] The surface of a CCD sensor is covered with 
pixels in a rectangular array, where each pixel consists 
of a photosensitive area and a non photosensitive area. 
The ratio of the photosensitive area to the total pixel size 
is called the "fill factor". This is usually less than 1 00%, 
typically 70-80%, but in some cases as low as 40%. The 
lower the value ( to a minimum of 25% ) the better as in 
that case, when we move the camera, we are actually 
seeing different object areas, as shown in Figs. 8(a) and 
8(b). 



[0037] The invention is not limited to the embodiments 
described. Forexample:- 

A: The resolution is not necessarily doubled. It could 
s be tripled by taking 9 images 1/3 pixel apart in X 
and Y. 

B: Colour images may be acquired simultaneously 
by switching the lighting colours between each 
10 move, such as:- 

Green [0, 0] 
Red[1, 0] 
Green [1,1] 
15 Blue [0,1] 

[0038] Thus, we could acquire colour and higher res- 
olution at the same time. However, chromatic aberration 
might reduce the accuracy in this case. 

C: It is not essential to acquire four images to double 
the resolution. The system may just acquire 2 im- 
ages at [ 0, 0 ] and [ HX, HY ], and interpolate the 
other two points at [ 0, HY] and [ HX, 0 ]. 
This would make the acquisition twice as fast as tak- 
ing 4 images, and would give an intermediate 
amount of detail, between taking 1 image and taking 
4 images. 

In addition, in this case, we can modify out standard 
horizontal and vertical vector reading algorithms, 
which read image regions N pixels wide to a linear 
vector of pixels, to only read the "real" pixels, and 
to ignore the interpolated pixels. 
In this case, the system can maintain the full reso- 
lution in X and Y, while only taking 2 images. The 
sequence is as shown in Fig. 9. 

[0039] Similarly, it is not essential to acquire 9 images 
to triple the resolution of the system. It could acquire 3 
images in a.diagonal, or 5 images in two diagonals. 
(TX = 1/3 X, TY= 1/3 Y ), and a sequence is illustrated 
in Fig. 10 for a 3 x 3 dense pattern. A 3 x 1 sparse pat- 
terns are shown in Figs. 11 and 12. 
[0040] It is also envisaged that the lens may be moved 
with respect of the camera, without moving the larger 
and heavier XY table. This would tilt the lens a little as 
follows: 

[0041] Suppose we are working at 2:1 reduction (i.e. 
the camera sensor pixel size is 12 microns and the ob- 
ject pixel size is 24 microns. 

Half an object pixel would be 12 microns, so if we as- 
sume that the camera and lens tilt linearly, we have to 
shift the focal point of the lens by 6 microns. This can 
easily be achieved by using a piezo electric mover on 
the lens clamp. The advantage of this is that the XY table 
does not need to be moved and the mass to be moved 
is much less. 

[0042] Alternately, the piezo mover could shift (rather 
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than bend) the lens to achieve this movement in an XY 
extension tube system. 

[0043] Also, the system can be used for "split axis" 
systems as well as XY table systems. A split axis system 
is one where one axis ( say X ) is used to move the cam- 
era ( al the "top" of the machine ) and the other axis ( say 
Y ) is used to move the object ( printed circuit board ) at 
the bottom of the machine. The principle is the same. 
[0044] Also, the image processing functions may 
"know" which views require higher ( 2x or 3x ) resolution 
images and which do not, and only performs the high 
resolution on the views which require it. The may be a 
function implementing an algorithm for deciding which 
components require higher resolution inspection based 
on their body width or lead width, to enable the system 
to mark which views require high resolution inspection. 
[0045] A function may read horizontal and vertical im- 
age vectors from the interpolated image, using only the 
"real" data. 

[0046] If the system is to acquire a sequence of imag- 
es with different characteristics ( for instance different 
coloured lighting or different angled lighting ), it does not 
have to acquire ail the images with 4 shifted images, it 
might only use 4 images for the most important image 
and use 2 images for the others. For instance, suppose 
we were acquiring 3 images ( Red, Green and Blue ) to 
generate a colour image, where the red images was the 
most important image ( primary image ), it might acquire 
the red image with 4 shifted images and the green and 
blue with two each, thus reducing the number of images 
acquired from 12 to 8. It might even be possible to ac- 
quire 4, 1 , 1 images to yield a total of 6 images. 
[0047] The Piezo mover could be a single axis mover, 
where the Piezo axis would be parallel to the heavy axis 
of the xy table. This is because the light axis of the xyta- 
ble is better at small moves than the heavy axis. Thus 
the light axis moves can be left to the xy table, while the 
heavy axis moves can be better done with a separate 
( Piezo ) mover. 

[0048] The invention is not limited to the embodiments 
described but may be varied in construction and detail. 



Claims 

1. An image processing system comprising a control- 
ler a camera, movement means to change effective 
location of the camera with respect to a target ob- 
ject, and means for processing the captured imag- 
es, wherein 

the camera and the movement means com- 
prise means for capturing a plurality of mutually 
offset initial images for a view, and 



initial images. 

2. A system as claimed in claim 1 , wherein the initial 
images are mutually offset by a fraction of a pixel 

5 width. 

3. A system as claimed in claim 2, wherein the initial 
images are offset by one half a pixel in X and Y di- 
rections. 

10 

4. A system as claimed in claim 2, wherein the initial 
images are offset by a third of a pixel in X and Y 
directions. 

*5 5. A system as claimed in any preceding claim, com- 
prising the further step of interpolating additional 
pixels from the captured initial images. 

6. A system as claimed in any preceding claim, where- 
20 jn the controller comprises means for determining 
for which parts of an object multiple offset initial im- 
ages are to be captured, whereby view images of 
differing resolutions are obtained for an object. 

25 7. a system as claimed in any preceding claim, where- 
in the controller comprises means for directing cap- 
ture of initial images of different colours for a partic- 
ular view image. 

30 8. a system as claimed in any preceding claim , where- 
in the movement means comprises means for mov- 
ing a lens with respect to the remainder of the cam- 
era. 

35 9. a system as claimed in claim 8, wherein the move- 
ment means comprises means for moving the lens 
in X and Y directions. 

10. A system as claimed in claims 8 or 9, wherein the 
40 movement means comprises means for tilting the 
lens. 
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the image processing means comprises means 
for combining the initial images to provide a 
view image of higher resolution than that of the 
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Step 1 Acquire first image 



Step 2 Acquire second Image 



Camera FOV 



1/2 Pixel Move 



Scene Under Examination 



Fig. 1(a) 



Fig. Kb) 



Step 3 Acquire third image 



Step 4 Acquire fourth image 





Fig. 1(c) 



Fig. 1(d) 
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Fig. 2(a) 
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Fig. 2(b) 
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Fig. 2(c) 



Fig. 2(d) 
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Step 5 Assemble Super Resolution Image 
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Fig. 3(a) 



Image Coordinate Reference 
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Step 1 Acquire first image 



Step 2 Acquire second image 



Camera FOV 



1/2 Pixel Move 




First FOV 



Second FOV 



Scene Under Examination 



Fig. 4(a) 



Fig. 4(b) 
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Fig. 5(b) 
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Step 3 Populate Super Resolution image 
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Fig. 6(a) 



Step 4 Interpolate missing pixels 
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Fig. 6(b) 



Legend 

Image 1 Pixel 



Image 2 Pixel [•] Interpolated Pixel 



11 



euj.Qnnr:tr>- «-pd 



EP 1 320 254 A1 



A: [ 0. 0 ] -> B: [ HX, 0] 
I 

V 

D: [HY, 0] <- C: [ HX, HY ] 



Fig. 7 



Hg. 8(a) 



Diagram Showing Smaller Photosensitive area 
within Geometric pixel rectangle. 



Fig. 8(b) 



A: [ 0, 0] 



C: [ HX, HY ] 
Sequence is A, C. 



Fig. 9 
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A:[ 0, 0]-> B:[TX, 0] -> C: [ 2Tx, 0] 

I 

V 

F:[ 0,Ty]<- E:[TX,Ty] <-D:[2Tx,Ty] 
I 

V 

G:[ 0, 0]-> H:[TX, 0] -> I: [ 2Tx, 0] 
Sequence ABCDEFGHI 

Fig. 10 

3 x 1 sparse diagonal pattern 
A:[ 0, 0] 

-> 

E:[TX,Ty] 

-> 

l:[2Tx, 0] 

Sequence: A, E, I 

Fig. 11 

3 x 1 sparse diagonal pattern 

A: [ 0, 0 ] -> -> C: [ 2Tx, 0 ] 

E:[TX,Ty] 

<- 

G:[ 0, 0]-> -> l:[2Tx, 0] 

Sequence: A, C, E, G, I 

Fig. 12 
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